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1.  INTRODUCTION 


Currently  the  U.S.  Army  is  developing  a system  for  the  acquisition, 
tracking,  and  laser  designation  of  military  targets  from  helicopters. 

The  Army  is  testing  a system  which  will  accomplish  automatic  hand-off  of 
targets  when  both  imaging  devices  are  day  TV  systems.  Very  little  atten- 
tion has  been  given  to  date  to  the  problem  of  hand-off  of  a target  when 
one  or  both  of  the  imaging  devices  are  active  in  different  spectra.  As  a 
result  the  U.S.  Army  Research  and  Development  Command,  Huntsville,  Alabama 
let  a contract  with  the  Engineering  Experiment  Station,  Auburn  University 
to  study  and  make  a recommendation  for  a system  to  accomplish  automatic 
hand-off  of  targets  from  designators  to  imaging  missile  seekers  when  the 
imaging  devices  are  dissimilar.  This  report  presents  the  results  of  that 
effort. 

A.  Scope  of  Work 

The  purpose  of  this  study  is  to  analyze  the  complexity  of  hardware 
and  software  required  to  implement  an  automatic  correlation  technique  to 
be  used  in  "handing  off"  a target  from  a target  acquisition  device  to  an 
Imaging  missile  seeker  ( IRIS  ).  The  function  of  an  automatic 
correlator  is  to  compare  the  video  signals  from  the  PTS  and  the  missile 
seeker  and  to  generate  error  signals  to  cause  the  missile  seeker  to  slew 
such  that  its  aimpoint  coincides  with  that  of  the  pointing  and  tracking 
unit.  The  work  shall  use  as  a starting  point  or  basis  the  results  obtained 
under  contract  DAAH01-76-C-0396  and  reported  in  [4].  Specifically  the 
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above  final  report  recommended  a correlator  implementation  for  hand-off 
from  a day  TV  PTS  to  a day  TV  seeker  system.  The  scope  of  work  performed 
under  this  contract  was  to  extend  the  above  results  such  that  automatic 
hand-off  of  targets  from  a day  TV  or  a Forward  Looking  Infra-Red  (FLIR) 
system  to  an  Infra-Red  Imaging  Seeker  (IRIS)  can  be  accomplished.  The 
scope  of  work  as  written  in  the  contract  is  given  in  the  following  five 
paragraphs  for  completeness. 

The  first  phase  shall  be  an  investigation  to  determine  as  many  tech- 
niques as  possible  for  preprocessing  the  two  images  obtained  from  sensors 
with  different  characteristics  such  that  hand-off  can  be  accomplished 
using  the  correlator  developed  in  contract  DAAH01 -76-C-0396.  The  emphasis 
shall  be  on  preprocessing  to  make  the  new  algorithm  compatible  with  the 
existing  correlator  rather  than  a complete  new  design  for  a correlator. 

The  purpose  of  a preprocessing  algorithm  is  to  convert  the  signals 
from  the  two  image  sensors  to  a compatible  form  such  that  correlation  can 
be  accomplished.  The  preprocessing  must  correct  for  differing  fields  of 
view  (FOV),  aspect  ratios,  frame  rates,  interlacing,  number  of  lines  per 
TV  frame,  signal  sampling  rates,  and  sensor  characteristics.  Sensor  charac- 
teristics are  divided  into  differing  spatial  and  spectral  responses. 

Spatial  differences  are  a result  of  the  detectors  used  in  each  sensor. 

For  example  a FLIR  may  have  a detector  array  or  a single  scanned  detector 

| 

while  standard  TV  has  a scanned  electron  beam.  Spectral  differences  are 
a result  of  the  detectors  being  sensitive  in  different  frequency  ranges 
such  as  in  the  visible  spectrum  and  the  IR  spectrum. 

Phase  Two,  which  will  actually  be  conducted  concurrently  with  Phase 
One,  shall  consist  of  an  investigation  of  the  hardware  and  software  needed 

— - J 
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for  various  combinations  of  PTS  and  missile  seeker  head  configurations. 
Consideration  shall  be  given  to  differing  characteristics  as  outlined 
above.  The  contractors  shall  consider  various  combinations  and  recommend 
the  hardware  and  software  necessary  to  implement  each. 

Phase  Three  shall  consist  of  the  actual  analysis  of  complexity  of 
the  hardware/software  necessary  to  implement  the  various  combinations  of 
Phase  Two.  This  analysis  shall  be  presented  in  such  a way  that  MIRADCOM 
will  be  able  to  use  it  to  make  a trade-off  of  the  various  options  avail- 
able. 

In  Phase  Four  a preprocessing  technique,  along  with  the  system  con- 
straints, shall  be  chosen  in  concert  with  the  MIRADCOM  technical  director. 
This  decision  will  be  based  on  the  results  of  the  first  three  phases. 

The  hardware  and  software  to  implement  this  approach  shall  be  completely 
specified,  and  the  preprocessing  algorithm  shall  be  developed. 

B.  Organization  of  Report 

All  of  the  work  outlined  in  the  Scope  of  Work  has  been  completed  and 
is  documented  in  this  final  report.  Chapter  two  formulates  the  prepro- 
cessing algorithms  investigated.  The  preprocessing  algorithms  given  in 
Chapter  two  convert  the  video  signals  received  from  the  imaging  system  to 
a digitized  binary  signal  based  on  the  measures  of  average  value,  stan- 
dard deviation,  or  edge  content. 

Chapter  three  outlines  a correlation  system  simulation  program  used 


to  study  the  preprocessors  given  in  Chapter  two.  The  results  of  a number 
of  simulations  are  presented  and  explained  in  Chapter  three.  The  results 
presented  in  Chapter  three  satisfy  task  three  of  the  scope  of  work. 
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Two  subjects  are  treated  in  Chapter  four.  First  a descriptive  re- 
view of  the  imaging  sensor  characteristics  is  given  in  terms  of  typical 
performance  parameters  and  their  interrelationships.  Secondly,  further 
analysis  of  threshold  statistics  for  the  binary  correlator  is  given  which 
seeks  to  place  the  analysis  on  a more  sound  theoretical  base  and  to 
include  the  very  real  possibility  of  pixel  mis-match  at  registration  due 
to  sensor  contrast  reversals  or  geometric  distortions.  A continuation 
of  previous  work  is  presented  for  completeness  which  allows  for  inclusion 
of  pixel  mis-match. 

Chapter  five  gives  the  conclusions  and  recommendations  resulting 
from  the  above  work  in  fulfillment  of  task  five  of  the  scope  of  work. 
Questions  which  arose  during  the  course  of  this  work  but  were  not  answered 
due  to  lack  of  time  or  lack  of  simulation  facilities  are  also  given. 


2.  FORMULATION  OF  PREPROCESSING  ALGORITHMS 
USED  TO  FORM  BINARY  IMAGES 


In  this  chapter  many  preprocessing  algorithms  are  presented  which  can 
be  used  to  transform  the  video  images  to  a binary  form  for  input  to  the 
correlation  process.  The  algorithms  discussed  in  sections  A and  B are 
applicable  to  a correlator  system  in  which  both  images  are  acquired  with 
sensors  sensitive  in  the  visual  spectrum.  The  algorithms  discussed  in 
section  C are  applicable  to  a correlator  system  in  which  the  HR  image 
is  acquired  through  a standard  TV  system  and  the  seeker  image  is 
acquired  with  an  infrared  imaging  seeker  (IRIS). 

A.  Quantization  Threshold  Based  on  Average  Value 

To  quantize  the  input  video  signals  to  one-bit  (2-level)  requires 
that  a threshold  value  be  set  such  that  all  signal  levels  above  and 
equal  to  the  threshold  value  become  1 and  all  signal  levels  below  the 
threshold  value  become  zero.  One  possible  way  to  set  the  threshold  is 
on  the  average  value  of  the  signal.  This  section  explains  two  procedures 
of  calculating  the  average  signal  value.  Both  methods  are  forms  of 
a varying  local  average.  This  is  done  so  that  real  time  operation 
can  be  achieved. 

1 . Line  Average  Quantizer 

Consider  a single  line  of  video  as  given  in  Figure  1.  The  pixel  to 
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be  quantized  is  X(i,  j).  The  line  average  quantizer  quantizes  the 
points  of  the  selected  input  array  by  calculating  the  average  value 
of  a specified  number  of  previous  pixel  values  of  the  same  line.  The 
quantization  threshold  is  then  this  average  value.  Equation  1 


ith 

row  1 


H 


T 


X(i,j) 


□ 


Figure  1.  Layout  for  quantization  based  on  a 
line  average. 


describes  the  average  value  calculation  where  L is  the  average  value 
sample  length. 

n L 

(1) 


X(i,j)  = f t,  X(i . j - J + 1) 
J=1 

Equation  2 gives  the  quantization  process. 


(1  ; X (i,j)  > X(i,j) 

,(i,j)  = { 

* 0 ; X (i,j)  < X(i,j) 


(2) 


where  X^  (•,*)  = quantized  pixel. 

This  process  would  continue  until  all  pixels  in  the  input  array 
are  quantized.  As  can  be  seen  from  Figure  1,  the  first  (L  - 1)  pixels 
of  each  video  line  will  be  quantized  incorrectly  and  must  be  ignored  n 
the  correlation  process.  If  the  original  input  array  is  of  size  N x M, 
then  the  quantized  array  will  contain  N x (M  - L + 1 ) valid  points. 
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2.  Array  Average  Quantizer 


Consider  the  layout  of  the  input  video  array  as  given  in  Figure 
2.  The  pixel  to  be  quantized  is  now  located  in  the  center  of  a K x L 
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[(K  - 1 )*M]  + L pixel  values  of  the  input  array  whereas  the  line 
average  quantization  process  would  require  (L  - 1 ) pixel  values  to  be 
saved. 

B.  Quantization  Threshold  Based  on 
Standard  Deviation 

A second  method  used  to  quantize  the  input  video  signals  to  one-bit 
is  explained  in  this  section.  Figure  3 illustrates  the  transfer  function 
of  the  quantizer.  This  algorithm,  developed  by  Goodyear  Aerospace  Corporation, 


-Ko  Ka 

Figure  3.  Bilevel  absolute  value  quantizer. 


uses  the  absolute  value  of  the  live  video  and  a threshold  value  proportional 
to  the  standard  deviation  of  the  video  signal  to  convert  the  live  video 
signal  to  a bilevel  digitized  array.  The  application  of  the  quantizer 
shown  in  Figure  3 requires  two  calculations  of  the  input  video.  First, 
the  average  value  must  be  removed  from  the  input  video  and  second,  the 
standard  deviation,  o,  of  the  input  video  must  be  obtained  so  the 
threshold,  Ka,  can  be  set.  The  average  value  can  be  estimated  by  either 
of  the  two  algorithms  given  in  part  A.  This  section  explains  two  algo- 
rithms used  to  estimate  the  standard  deviation  of  the  video  signal. 
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1 . Standard  Deviation  of  a Line  Sample 

Consider  a single  video  line  as  shown  in  Figure  1.  The  sample 
standard  deviation  at  the  (ith,  jth)  pixel  based  on  a sample  length  L 
is  given  by  Equation  4. 


a(i,j)  = 


L 

U S {x(i,j-J+l)  - *(i,j)}2 

J=1 


1/2 


(4) 


Then,  expanding  the  term  on  the  right  yields 


>(i.j)  = f 


L J=1 


X2(i,j-J+1)  - 2 X(i,j-J+1)  X( 


i.j)  + X^(i  ,j)| 


1/2 


a(i,j)=[l  X2(i,j-J+1)  - 2X(i,j)  £ X(i,j-J+1)  +X2(i,j)}l 

L J=1  (5)  J 


1/2 


Substituting  Equation  1 into  5 yields 

L 


o( 


ijj  = l ^ X2(i,j-J+1)  - X2(i,j) 


J=1 


1/2 


(6) 


The  threshold  value  is  proportional  to  the  standard  deviation  calculated 
by  Equation  6.  Equation  7 gives  the  quantization  process. 


Xq(i.j)  - 


1 ; X(i,j)  > Ko(i,j)  + X(i,j) 

0 ; X(i,j)  - Ko(i.j)  < X(i,j)  < Ko(i,j)  + X(i,j)  (7) 

1 ; X(i,j)  < X(i,j)  - Ko(i,j) 


K,  given  in  Equation  7,  is  a positive  proportionality  constant  which 
would  have  to  be  known  a-priori. 
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2.  Standard  Deviation  of  an  Array  Sample 

Consider  the  K x L array  sample  of  the  input  video  as  shown  in  Figure 
2.  The  sample  standard  deviation  based  on  the  K x L subarray  is  given  by 
Equation  8. 

" 1/2 


a(i,j)  = 


K L 


4l  22  {X(i“k-1+I*  - X(i,j)p 


(8) 


1=1  J=1 

where  X(i,j),  k,  and  l are  given  by  Equation  3.  Then,  expanding  the 
term  on  the  right  yields. 


o(i,j)  = 


K L 


^^2  X2(i-k-l+I,  j-  l-  1+J)  - X2( 


i ,j)j 


1/2 


(9) 


1=1  J=1 

The  quantization  process  would  again  be  given  as  in  Equation  (7). 
C.  Quantization  Threshold  Based  on  Edge  Content 


In  order  for  the  correlation  of  binary  images  to  be  a valid  measure 
of  similarity  it  is  necessary  that  the  quantization  process  be  based  on 
similar  measures  of  scene  content.  For  the  case  in  which  the  images  are 
obtained  from  sensors  operating  in  different  frequency  spectrums,  quantiza- 
tion based  on  average  value  or  standard  deviation  no  longer  supplies  similar 
binary  images  to  the  correlation  system.  Therefore,  some  other  measure 
on  which  quantization  is  based  must  be  used  when  correlating  images  from 
dissimilar  sensors.  One  such  measure  investigated  in  this  contract  is 
that  of  edge  content  in  the  two  images.  All  edge  detection  schemes  base 
their  decision  on  the  gradient  value  associated  with  a particular  pixel 
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in  the  image.  Pixels  exhibiting  a high  gradient  value  as  compared  with 
a threshold  are  considered  to  be  edge  points  and  those  pixels  with  low 
gradient  value  are  considered  non-edge  points. 

This  section  explains  two  edge  detection  algorithms  that  could  be 
implemented  as  a preprocessing  step  in  the  correlation  system.  The 
algorithms  are  basically  two  different  ways  to  estimate  the  gradient 
magnitude  at  a particular  pixel  of  interest  with  no  preference  given  to 
direction  of  the  gradient.  It  should  be  noted  that  both  methods  require 
selection  of  a gradient  threshold  to  form  binary  images.  The  problem  of 
selecting  a proper  gradient  threshold  will  be  addressed  in  a later  chapter. 


1 . 2x2  Edge  Detector 


The  method  of  edge  detection  presented  here  is  known  as  the  Roberts 
Cross  Operator  [1].  This  method  is  based  on  the  principle  of  estimating 
and  selecting  areas,  in  the  digital  picture,  of  high  gradient  activity. 
Figure  4 is  the  pixel  representation  of  a 2 x 2 area  in  the  digital 
picture.  If  the  digital  picture  is  represented  by  the  two-dimensional 
function  g(x,y),  then  the  magnitude  of  the  gradient  at  pixel  (i,j)  can 
be  approximated  by  Equation  10.  From  Equation  10  it  can  be  seen 


(i,j) 

\ 

(i,j+l) 

/ 

✓ 

(i+1  »j) 

(i+l.j+1) 

Figure  4.  Pixel  representation  for  the  Roberts 
cross  operator. 


II?  g(i.j)ll  $ R(i,j)  = {[g(i,j)  - g(i+i.j+i)r  + [g(i.J+D 

o 1/2 

- g(i+i.j)]z)  (io) 
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that  in  picture  areas  of  constant  grey  level,  R(i,j)  will  be  zero  and  in 
picture  areas  of  high  grey  level  change  in  either  the  X or  Y or  both 
directions,  R(i,j)  will  be  large.  A more  computationally  efficient 
algorithm  is  given  in  Equation  11.  F(i,j)  behaves  very  much  like  R(i,j). 

F(i.j)  = |g(i,j)  - g(i+i,j+l)|  + |g(i,j+i)  - g(i+l,j)|  (n) 

To  implement  binary  quantization  using  the  gradient  properties  of 
Equation  11  necessitates  the  selection  of  a gradient  threshold  for  values 
of  F(x,y).  The  quantization  process  is  given  in  Equation  12,  where  Fq(.,.') 
is  the  quantized  gradient  array  and  GTH  the  gradient  threshold.  The 


j * 

effect  if  applying  Equations  11  and  12  to  the  original  digital  picture  is 
an  outline  or  edge  drawing  of  objects  or  areas  of  constant  grey  level 
found  in  the  picture. 

2.  3x3  Edge  Detector 

The  estimation  of  the  gradient  magnitude  at  a particular  pixel  can 
be  extended  to  include  the  effects  of  the  eight  nearest  pixels  surrounding 
the  pixel  of  interest.  Figure  5 illustrates  a 3 x 3 pixel  area  used  to 


(i-U-1) 

(1-1  J) 

(i-1,3+1) 

(i.j-1) 

(i,j) 

(i.j+1) 

(i+lj-l) 

(1+1  »j) 

(i+1 .j+1 ) 

1 ; F(i,j)>GTH 
0 ; F ( i , j ) < GTH 


(12) 


Figure  5. 


Pixel  representation  for  the  3x3 
edge  operator. 
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estimate  the  gradient  magnitude  at  pixel  (i,j).  The  operation  of  calculating 
the  gradient  magnitude  can  be  defined  using  an  inner  or  dot  product 
formulation.  Assume  that  the  digital  picture  function  is  given  by 
G(x,y)  and  an  appropriate  weighting  function  by  W(x,y),  where  W is 
a 3 x 3 matrix.  Then  the  dot  product  of  G(x,y)  with  W(x,y)  is  given  by 
Equation  13. 

3 3 

■ EE  G(i+K-2,  j+L-2)  W(K,L)  (13) 

K=1  L=1 

A number  of  weighting  matrices  can  be  found  in  the  literature  [2], 

Figure  6 illustrates  three  sets  of  weighting  matrices.  The  magnitude 
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b.  Sobel  weighting  functions 
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c.  Isotropic  weighting  functions 
6.  Weighting  matrices  used  for  gradient  estimation. 
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of  the  gradient  at  pixel (i,j)  can  be  estimated  as  follows. 
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Let 


Sx(i,j)  = (G,  W] ) and 

Sy(i,j)  - (G,  W2) 


04) 


Then  the  gradient  magnitude  is  given  by  Equation  15 

I | vg(i  ,j ) | | = [Sx2-(i,j)  + Sy2(i,j)J  (15) 

A more  computationally  efficient  algorithm  is  given  in  Equation  16  and  the 
quantization  process  would  proceed  as  given  in  Equation  (12).  The  most 
F(i,j)  = |Sx(i,j)|  + | Sy(i ,j ) | (16) 

common  of  the  three  weighting  matrices  is  that  due  to  Sobel  and  was 
chosen  as  the  3x3  edge  detector  for  studies  made  in  this  report. 

As  can  be  seen  from  Figures  4 and  5 the  binary  image  arrays  are 
somewhat  reduced  in  size  compared  to  the  original  input  image  array. 

If  the  original  image  is  of  size  N x M,  then  the  2x2  edge  operator 
produces  a binary  array  of  dimension  (N-l)x(M-l)  and  the  3x3 
edge  operator  produces  a binary  image  of  dimension  (N  - 2)  x (M  - 2). 

In  general  if  the  edge  detection  operation  is  based  on  n x m pixels,  the 
resulting  binary  array  will  be  of  dimension  (N  - n + 1 ) x (M  - m + 1 ). 
These  reductions  of  array  size  present  no  problems  in  the  correlation 
system. 
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3.  Other  Edge  Detectors 

There  are  many  other  types  of  edge  detection  algorithms  found  in 
the  literature.  In  most  cases  they  are  extensions  of  the  3x3  edge 
detector  to  include  the  properties  of  gradient  direction  and  edge  point 
connectivity.  These  edge  detection  schemes  were  not  implemented  in  the 
correlation  system  study  since  they  would  impose  more  stringent  storage 
and  computation  time  requirements  on  the  system.  However,  a brief  outline 
of  how  these  systems  operate  will  now  be  given.  Figure  7 shows  in  block 
diagram  form  a more  elaborate  edge  detection  system  [3].  In  this  system 
the  decision  to  form  the  binary  image  is  no  longer  conditioned  on  threshold 
only,  but  on  a combination  of  gradient  activity,  threshold,  and  connectivity 
of  the  edge  direction  to  other  pixels.  The  gradient  picture  is  formed  by 
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Figure  7.  A more  elaborate  edge  detection  system. 

selecting  the  maximum  magnitude  produced  by  any  one  of  the  sets  of  masks 
shown  in  Figure  8.  The  direction  of  the  gradient  is  chosen  by  which 
mask  produces  the  maximum  magnitude  output.  Figure  9 illustrates  the 


I 


22 


Direction  of  Direction  of  Prewitt  Kirsch  Three-level  Five-level 
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Figure  8.  Directional  weighting  matrices. 
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principal  directions  selected.  It  should  be  noted  from  Figure  8 that 


Figure  9.  Principle  directions  of  edges. 

in  the  case  of  the  three-level  and  five-level  simple  masks  only  the 

gradient  value  of  the  first  four  masks  need  be  calculated.  The  direction 

of  the  edge  is  then  given  by  the  sign  of  the  gradient  value  of  the  mask 

producing  the  maximum  absolute  gradient  value.  The  edge  directions  at 

each  pixel  generate  the  edge  direction  map  shown  in  Figure  7.  The  threshold 

map  shown  in  Figure  7 is  normally  produced  by  a locally  adaptive  process 

such  as  the  one  given  in  [3].  This  method  is  given  in  Equation  17,  where 

LAT  is  the  locally  adaptive  threshold. 

_ MAX  ( 1 Ykl  , k = 0,1,. ..,7} 

Output  of  the  low  pass  filter  MQ  at  pixel  (i,j) 

where  Y^,  k = 0,1,..., 7 are  the  gradient  values  at  pixel  (i,j) 

1 2 1 

2 4 2 

1 2 1 

The  process  of  Equation  17  is  one  of  comparing  the  gradient  image  with  a 

k 

k 
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blurred  version  of  the  original  image  produced  by  the  low-pass  operation 
in  the  denominator. 

In  determining  if  there  is  an  edge,  a local  connectivity  test  is 
performed  using  the  edge  direction  map.  One  such  connectivity  test  is 
shown  in  Figure  10.  The  preceding  and  succeeding  edge  directions  must 
satisfy  one  of  the  possibilities  shown  in  Figure  10  for  the  test  to 
succeed.  If  the  connectivity  test  succeeds  and  the  gradient  value 
is  above  the  threshold,  then  an  edge  point  has  been  found.  Thus,  the 
binary  edge  map  is  generated  at  the  output. 


Figure  10.  Local  connectivity  test  of  edge 
directions. 


3.  SIMULATION  RESULTS  OF  THE 
CORRELATION  SYSTEM 


This  chapter  describes  the  simulation  program,  developed  by  contract 
personnel,  used  to  study  the  effects  of  the  preprocessors  described  in 
Chapter  2.  Section  A provides  a general  explanation  and  block  diagram 
of  the  simulation  program.  A complete  listing  and  detailed  flow  charts 
of  the  simulation  program  are  compiled  as  a supplement  to  this  report  and 
can  be  obtained  from  the  project  director.  This  chapter  also  presents 
explanations  and  results  of  studies  conducted  using  the  correlation  sys- 
tem simulation  program. 

A.  The  Simulation  Program 

In  order  to  analyze  the  effects  on  the  correlation  system  of  the 
various  preprocessors  given  in  Chapter  2,  it  was  necessary  that  a gener- 
alized correlation  system  simulation  program  be  written.  Figure  11  il- 
lustrates a correlator  hand-off  system  as  proposed  in  [4].  The  simula- 
tion program  is  structured  to  perform  all  functions  illustrated  by  Figure 
11  through  the  calculation  of  the  correlation  array.  Figure  12  shows  in 
block  diagram  form  the  functions  performed  by  the  simulation  program. 

The  simulation  program  is  coded  using  the  FORTRAN  language  and  structured 
in  subroutine  format  for  ease  in  substitution  or  modification  of  algorithms. 

A general  description  of  the  operation  of  the  program  will  now  be 
given. 
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Figure  11.  Block  diagram  of  correlator  hand-off  system. 
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1 . Image  Input 

The  simulation  program  receives  as  input  digitized  images  of  the 
scene  being  simulated.  The  digital  images  are  stored  in  TV  field  arrays 
on  magnetic  tape.  The  magnetic  tape  is  structured  in  file/record  for- 
mat. Each  magnetic  tape  file  contains  two  TV  fields,  with  each  field 
consisting  of  1024  sequential  magnetic  tape  records.  Each  record  con- 
sists of  120  pixel  values  ranging  from  0 to  225,  with  255  indicating 
black  and  0 indicating  white.  Thus  successive  pairs  of  odd  and  even 
numbered  records  compose  one  column  in  the  digital  picture  array.  Figure 
13  shows  the  grouping  of  the  magnetic  tape  records  to  yield  the  two  TV 
fields  from  each  magnetic  tape  file.  As  can  be  seen  from  Figure  13  the 
simulation  program  can  use  field  data  arrays  (240  x 512)  or  the  two 
fields  can  be  interlaced  by  rows  resulting  in  frame  data  arrays  (480  x 512). 
For  the  simulation  results  which  follow,  the  field  data  arrays  were  used 
as  the  input  images.  Fields  were  used  because  of  the  large  amount  of 
memory  required  in  a real-time  correlator  which  correlates  on  frames 
rather  than  fields. 

2.  Resolution  Reduction 

In  order  that  the  two  images  possess  the  same  spatial  resolution 
characteristics,  it  is  necessary  that  the  resolution  of  the  HR-TV  image 
be  reduced  to  that  of  the  sensor  image.  The  resolution  reduction  algo- 
rithm accomplishes  this  task.  The  algorithm  performs  the  calculations  of 
Equation  67  given  in  [4].  The  algorithm  compensates  for  differences  in 
resolution  between  the  HR-TV  image  and  the  sensor  image  caused  by  differ- 
ing fields  of  view,  number  of  TV  lines  per  frame,  frame  rate,  aspect  ratio. 
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, 1 

and  sampling  rate  of  the  two  imaging  systems.  These  parameters  are  used 
by  the  algorithm  to  calculate  the  vertical  (Wv)  and  horizontal  (WH) 
scale  factors  needed  to  make  the  resolution  adjustment  of  the  HR-TV  image. 

Depending  on  the  type  of  sensor  image  being  used  the  scale  factors  would 
change  appropriately. 

3.  Reference  Extraction 

The  reference  extraction  algorithm  removes  from  the  HR-TV  image  an 
array  of  size  K X L to  be  used  as  the  reference  image.  The  reference 
size  and  extraction  coordinates  in  the  HR-TV  image  are  input  parameters 
for  this  algorithm. 

4.  Binary  Quantization 

The  next  operation  performed  in  the  simulation  program  is  that  of 
binary  quantization.  One  of  the  preprocessors  of  Chapter  2 would  be 
implemented,  depending  on  the  type  of  simulation  being  run. 

5.  Correlation  Calculation 


At  this  stage  in  the  simulation  program  a mean  absolute  difference 
(MAD)  calculation  is  made  for  each  placement  of  the  reference  image  in 
the  sensor  image,  thereby  generating  a complete  correlation  array.  The 
maximum  correlation  value  and  its  coordinates  in  the  sensor  image  are 
identified. 

B.  Photographs  of  the  Video  Images 
Used  in  the  Simulations 


In  this  section  photographs  of  the  video  images  used  in  the  simula- 
tion program  are  listed.  These  photographs  are  included  to  give  the 
reader  some  idea  as  to  what  types  of  scenes  were  tested.  The  simulation 


results  given  in  the  next  section  will  refer  to  the  scenes  as  they  are 

numbered  in  this  list.  The  abbreviations  used  in  the  titles  on  the 

photographs  are  as  follows: 

NFOV  - Narrow  field  of  view 
WFOV  - Wide  field  of  view 
IRIS  - Infra-Red  Imaging  Seeker 
WH  - White  Hot 
TV  - Television 

Also  included  in  this  section  are  computer  generated  density  and  edge 
plots  of  Scene  6.  Figure  29(a)  is  a computer  generated  image  of  Scene  6 
NFOV-TV  with  a vertical  and  horizontal  scale  factor  of  4 where  the  grey 
scale  data  has  been  reduced  to  10  shades.  Figures  29(b)  through  29(g) 
are  plots  of  the  2x2  and  3x3  edge  detection  algorithms  using  three 
threshold  levels  for  edge  discrimination.  Figure  30  and  31  are  the  binary 
images  produced  by  the  2x2  and  3x3  edge  detection  algorithms  using  the 
HR-TV  and  IRIS-WH  images.  The  HR-TV  horizontal  and  vertical  scale  factors 
used  were  4.722  x 18.0,  respectively.  The  edge  plots  are  shown  for  three 
different  gradient  threshold  settings  to  give  the  reader  an  idea  of  the 
difference  produced  by  the  two  edge  detection  algorithms. 

C.  Results  of  the  Average  Value  Preprocessors 
In  this  section  the  results  of  simulations  carried  out  using  the 
average  value  preprocessors  are  presented.  While  the  main  thrust  of  this 
contract  was  to  find  a correlation  system  configuration  to  handle  TV  to  IR 
Correlations,  TV-to-TV  simulations  were  attempted  to  support  to  results 
given  in  [4].  Table  1 illustrates  the  simulation  results  of  TV-to-TV 
correlations  for  scenes  1 to  4.  As  given  in  Table  1 the  correlations 
were  implemented  by  extracting  the  reference  array  from  the  High  Resolution 

I. 
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Figure  15.  Scene  1 WFOV  - TV  - NASA  tower. 
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Figure  16.  Scene  1 IRIS  - WH  - NASA  tower. 


Figure  18.  Scene 


Figure  19.  Scene  3 NFOV  - TV  - Woods. 


Figure  22.  Scene  4 IRIS  - WH  - Parki 


Figure  23.  Scene  5 NFOV  - TV  - Jeep  in  Parking  lot. 


Figure  25.  Scene  6 NFOV  - TV  - Jeep  in  front  of  fence. 


Figure  26.  Scene  6 IRIS  - WH  - Jeep  in  front  of  fence. 
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a.  Ten-level  density  plot. 

Figure  29.  Computer  generated  images  for  Scene  6-NFOV-TV  with  WH=4.0 
and  Wv=4.0 
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Figure  29b.  Output  of  2 x 2 edge  detector  with  gradient  threshold=10.0 
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Figure  29c.  Output  of  2 x 2 edge  detector  with  gradient  threshold* 30.0 
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Figure  29d.  Output  of  2 x 2 edge  detector  with  gradient  threshold=50.0 
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Figure  29e.  Output  of  3 x 3 edge  detector  with  gradient  threshold=50.0 
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Figure  29f.  Output  of  3 x 3 edge  detector  with  gradient  threshold3 100.0 
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Figure  29g.  Output  of  3 x 3 edge  detector  with  gradient  threshold=150.0 
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Figure  30d. 
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(HR)  - TV  image.  This  reference  was  then  matched  to  that  of  the  Low- 
Resolution  (LR)  - TV  image.  The  estimated  match  coordinates  in  the  LR 
image,  given  by  Est.  IMAX  and  Est,  JMAX  in  the  table,  were  found  by 
visually  inspecting  density  plots  of  the  images  under  test.  The  values 
Cal.  IMAX  and  Cal.  JMAX  were  found  using  the  computer  simulation.  The 
value  Cal.  RMAX  is  the  maximum  correlation  value  computed.  In  all  cases 
given  in  Table  1 the  reference  size  was  chosen  to  be  31  x 31  yielding  a 
possible  maximum  correlation  value  of  961.  The  line  average  sample 
length  was  set  at  11  and  the  array  average  sample  size  was  set  at  11  x 11. 

As  can  be  seen  from  Table  1 the  estimated  and  calculated  match 
coordinates  are  very  close  with  a difference  of  only  a few  pixels  in 
most  simulations.  This  indicates  that  the  line  average  and  array  aver- 
age preprocessors  work  well  for  TV-to-TV  correlations  carried  out  on 
images  with  a wide  variety  of  spectral  characteristics.  Cross-sectional 
plots  of  the  correlation  function  generated  through  the  maximum  value 
along  the  I and  J axis  also  give  some  insight  into  the  "goodness"  of  the 
preprocessors.  Figures  32  through  35  show  the  cross-sectional  plots  from 
simulations  made  on  scenes  1 and  4 in  Table  1.  These  plots  show  that  the 
line  average  quantizer  produces  a sharper  correlation  peak  than  does  the 
array  average  quantizer.  The  more  well  defined  correlation  peak  was  also 
evident  in  the  cross-sectional  plots  of  the  other  simulations  using  the 
line  average  quantizer  listed  in  Table  1. 

The  next  set  of  simulations  tried  were  those  in  which  the  sensor 
image  was  acquired  using  the  IRIS  and  the  average  value  preprocessors 
were  used  for  binary  conversion.  The  results  of  these  simulations  are 
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Figure  32.  Cross-sectional  plots  of  scene  1 correlation 
using  the  line  average  quantizer. 
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Figure  33.  Cross-sectional  plots  of  scene  1 correlation 
using  the  array  average  quantizer. 
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Figure  34.  Cross-sectional  plots  of  scene  4 correlation 
using  the  line  average  quantizer. 
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Figure  35.  Cross-sectional  plots  of  scene  4 correlation  using 
the  array  average  quantizer. 
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given  in  Table  2.  Due  to  the  dynamic  response  of  the  IRIS,  which  will 
be  explained  in  a later  section,  the  reduced  HR-TV  array  is  of  size 
50  x 28  pixels,  thus  the  reference  size  is  reduced  considerably  in  the 
J axis.  For  the  simulations  presented  in  Table  2,  the  line  average 
sample  length  was  set  at  7 pixels  and  the  array  average  sample  size  was 
set  at  7 x 7 pixels. 

As  illustrated  in  Table  2 there  are  large  differences  in  the  esti- 
mated and  calculated  match  coordinates  for  either  the  I or  J or  both 
coordinates  for  all  simulation  runs.  Also  given  in  Table  2,  and  Tables 
3 through  5,  are  the  five  highest  correlation  peak  values.  These  values 
are  included  to  see  if  the  true  match  location  occurred  at  a peak  less 
than  the  maximum  peak  condition.  There  is  only  one  case  in  Table  2 where 
this  is  possibly  the  situation.  In  the  run  for  scene  4 using  the  array 
average  quantizer,  the  fourth  peak  value  possesses  coordinates  close  to 
the  estimated  values.  However,  none  of  the  other  simulations  show  this 
tendency. 

Because  of  the  inability  of  the  line  average  and  array  average  quan- 
tizers to  produce  any  valid  TV-to-IR  correlations  for  the  simulations 
presented  in  Table  2,  these  preprocessors  would  not  be  viable  canidates 
to  be  used  for  TV-to-IP.  correlation.  It  is  possible  though,  that  the 
average  value  preprocessors  may  produce  better  results  when  operating  on 
characteristics  of  the  image  other  grey  scale  differences.  The  results 
given  in  Table  2 and  discussed  above  led  to  the  use  of  the  edge  detector 
preprocessors  for  TV-to-IR  correlation  studies. 
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D.  Results  of  the  Standard  Deviation  Preprocessors 

The  third  set  of  simulations  attempted  were  those  of  TV-to-IR 
correlations  using  the  standard  deviation  preprocessors.  Table  3 shows 
the  results  of  those  simulations.  Again  the  line  sample  length  was  7 
and  the  array  sample  size  was  7x7.  The  0 - 1 quantization  threshold 
was  set  at  lo  above  and  below  the  average  value.  The  results  shown  in 
Table  3 are  very  poor.  In  none  of  the  simulations  do  the  first  five  peak 
values  show  coordinates  close  to  the  estimated  coordinates.  There  are  a 
number  of  factors  not  illustrated  in  Table  3 which  could  effect  the 
correlation  results  due  to  these  preprocessors.  These  factors  are: 

(1)  sample  size  for  the  average  value  and  .standard  deviation  calculations; 

(2)  proportionality  constant,  K,  chosen  for  the  Ka  quantization  thresh- 
old; (3)  size  of  the  reference  image.  Due  to  time  limitations,  these 
factors  were  not  studied  thoroughly  but  should  be  considered  in  more 
deatil  before  the  standard  deviation  preprocessors  are  eliminated  for 
TV-to-IR  correlation.  The  conclusion  here  is  that  from  the  results  of 
Table  3 it  is  highly  unlikely  that  the  standard  deviation  preprocessors 
would  be  able  to  accomplish  TV-to-IR  correlation  without  considerably 
more  study  of  the  factors  which  effect  the  correlation  function. 

E.  Results  of  the  Edge  Detector  Preprocessors 
1 . Simulation  Results 

Since  the  average  value  and  standard  deviation  preprocessors  exhib- 
ited poor  performance  for  TV-to-IR  correlation,  a major  portion  of  the 
work  performed  for  this  contract  was  involved  in  simulations  for  TV-to-IR 
correlations  using  the  edge  detector  preprocessors.  A number  of  tests 


were  carried  out  with  edge  detector  preprocessors  for  the  various  scenes 
given  in  section  B of  this  chapter.  The  results  of  these  tests  are 
given  in  this  section. 

In  Table  4 the  results  of  tests  using  scenes  1 and  4 show  some 
preliminary  results  obtained  with  the  edge  detector  preprocessors.  In 
Tables  4 and  5,  TVGTH  refers  to  the  gradient  threshold  used  to  convert 
the  HR-TV  gradient  image  to  a binary  image  and  IRGTH  is  the  threshold 
value  used  for  the  IRIS  image.  Run  numbers  1 and  2 of  Table  4 were 
made  to  show  the  effect  of  reference  size  on  the  shape  of  the  correlation 
surface.  Figure  36  shows  the  cross-sectional  plots  of  the  correlation 
surface  for  Run  No.  1.  At  the  time  of  this  simulation  the  wide  side 
peaks  in  the  J-axis  cross  section  caused  some  concern.  The  scene  for 
this  run  was  reviewed  and  it  was  found  that  the  particular  reference 
chosen  contained  a large  area  of  zeroes.  It  was  felt  that  the  large 
area  of  zeroes  were  matching  similar  zero  areas  to  the  left  and  right 
sides  of  the  NASA  tower  in  the  IR  edge  image.  To  check  this  effect  the 
reference  image  was  reduced  to  a size  of  16  x 20  which  contained  approxi- 
mately half  one  values  and  half  zero  values.  It  was  reasoned  that  if 
the  matching  on  the  zeroes  was  occurring  then  the  side  peaks  should  be 
reduced  considerably.  Figure  37  shows  the  cross-sectional  plots  for 
this  test.  A comparison  of  Figures  36  and  37  shows  that  the  wide  side 
peaks  in  the  J-axis  completely  disappear  and  the  sharpness  of  the  peak 
in  the  I-axis  plot  was  improved.  This  test  confirmed  the  above  specula- 
tion and  also  gave  the  first  clue  as  how  to  select  the  zero-one  thresh- 
old for  the  HR-TV  gradient  image.  Runs  3 and  4 were  made  to  see  if  the 
correlation  function  would  track  a displacement  made  in  the  selection  of 
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Figure  36.  Cross-sections  of  Run  1 - Table  4 through 
the  maximum  value. 
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Figure  37.  Cross-sections  of  Run  2 - Table  4 
through  the  maximum  value. 


75 


the  reference  image.  As  shown  by  the  estimated  IMAX  coordinates  the 
reference  was  displaced  by  6 rows  in  the  two  runs.  The  calculated  IMAX 
coordinates  reflect  a change  of  4 rows  with  no  change  in  the  JMAX 
coordinates. 

Runs  5 through  10  or  Table  4 were  made  to  illustrate  the  effects 
of  changing  IRGTH.  The  gradient  threshold  used  for  the  HR-TV  image  was 
set  to  yield  approximately  an  equal  distribution  of  l's  and  0's  in  the 
reference.  Run  6 shows  that  a false  peak  situation  did  occur,  while  the 
second  and  third  peak  correlation  values  illustrate  true  match  conditions 
for  this  simulation. 

Runs  5 through  10,  excluding  run  6,  also  give  an  indication  as  to 
the  reliability  of  the  2x2  and  3x3  edge  detection  algorithms.  The 
reliability  of  these  algorithms  is  illustrated  by  the  difference  in 
magnitude  of  the  true  correlation  peak  and  the  second  and  third  peak 
correlation  values.  The  difference  in  successive  peak  correlation  values 
with  the  first  due  to  the  3x3  edge  detector  are  much  larger  than  those 
produced  by  the  2x2  edge  detector.  Due  to  this  fact,  a false  correla- 
tion peak  would  less  likely  occur  when  using  the  3x3  edge  detector,  but 
erroneous  edge  points  caused  by  noise  in  the  image  could  possibly  cause 
false  correlation  peaks  when  using  the  2x2  edge  detector.  As  shown  in 
section  B of  this  chapter  the  3x3  edge  detector  tends  to  widen  edges 
in  the  image  which  causes  a smoothing  effect  on  the  edge  points  due  to 
stray  noise.  The  2x2  edge  detector  does  not  exhibit  edge  point  smooth- 
ing and  edge  points  caused  by  noise  would  propagate  through  the  correla- 
tion system. 
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Table  5 shows  the  simulation  results  for  tests  made  on  scenes  6 and 
7.  In  these  runs  the  reference  array  was  extracted  from  three  different 
row  coordinates  in  the  complete  HR  image.  In  each  case  the  HR-TV  thresh 
old  remained  constant  and  IRGTH  was  adjusted  to  yield  the  highest  correla- 
tion value.  The  process  for  adjusting  the  IRIS  threshold  will  be  ex- 
plained later.  As  can  be  seen  from  Table  5 the  results  indicate  good 
correlations  in  most  runs.  Run  4 shows  a multiple  peak  correlation  value. 
The  first  and  second  peak  correlation  values  are  equal  in  magnitude,  one 
indicating  proper  match  coordinates  and  the  other  a false  peak  condition. 

A slight  adjustment  of  IRGTH  would  probably  rectify  this  result. 

In  this  section  the  results  of  a number  of  correlation  simulations 
have  been  presented  which  show  that  TV-to-IR  correlation  can  be  accom- 
plished using  edge  detection  algorithms  for  a number  of  different  scenes. 
However,  there  were  cases  in  which  correlations  were  attempted  and  no 
favorable  results  could  be  obtained.  Scene  5 is  one  example.  The  edge 
plots  produced  for  the  HR-TV  and  IR  images  showed  very  little  resemblance. 
Correlations  were  attempted  with  very  poor  results.  The  overall  conclu- 
sion from  all  the  simulations  was  that  edge  detection  preprocessing  of 
the  video  did  lead  to  correct  correlations  most  of  the  time  whereas  other 
algorithms  failed  most  if  not  all  of  the  time.  It  was  also  concluded 
from  the  simulations  that  proper  selection  of  the  gradient  threshold  is 
critical.  Methods  to  select  the  gradient  thresholds  are  discussed  in 
the  next  section. 

2 . Determination  of  the  Gradient  Threshold  for  the  HR-TV  Image 

From  a study  reported  in  [5],  it  was  determined  that  the  quantized 
HR-TV  reference  image  should  contain  an  equal  number  of  edge  points  (l's) 


Table  5.  Results  of  TV-to-IR  correlations  using  the 

edge  detector  preprocessor  on  Scenes  6 and  7. 
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and  nonedge  points  (0's).  The  ratio  of  l's  and  0's  in  the  reference 
image  is  dependent  on  the  value  of  gradient  threshold  selected.  Figure 
38  is  a rough  plot  which  demonstrates  how  the  number  of  l's  and  Os  in 
the  reference  image  vary  with  threshold.  As  the  plot  indicates  a 50-50 
mix  of  l's  and  0's  can  be  obtained  if  the  median  gradient  value  of  the 
generated  HR-TV  gradient  array  is  used  as  the  threshold.  The  above 
requires  storage  of  the  gradient  array  to  identify  the  median  gradient 


Figure  38.  1-0  Distribution  vs.  Threshold 


value.  Then  quantization  of  the  gradient  array  could  proceed.  This 
method  of  theshold  selection  for  the  reference  image  was  used  for  a 
number  of  the  simulations  given  in  Tables  4 and  5 with  favorable  results. 
3 . Determination  of  the  Gradient  Threshold  for  the  IRIS  Image 

In  order  to  determine  the  best  or  optimum  threshold  to  be  used  for 
the  IR  image  a number  of  simulations  were  made  by  these  investigators  and 
by  the  investigators  as  reported  in  [5]  to  study  the  sensitivity  of  the 
correlation  peak  to  IR  threshold.  The  optimum  threshold  is  defined  as 
that  which  yields  the  highest  correlation  value  given  a fixed  reference 
image.  Figure  39  is  a plot  of  maximum  correlation  values  vs.  threshold 


Figure  39.  Peak  correlation  value  vs.  IRIS  threshold  for  scene  4 - parking  lot. 
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using  data  of  scene  4.  Most  of  the  scenes  exhibited  curves  of  the  same 
shape.  This  curve  shows  that  the  peak  correlation  value  is  very  insensi- 
tive to  values  of  threshold  in  the  range  of  65  to  95.  In  this  threshold 
range  the  peak  correlation  value  deviates  from  the  maximum  value  by  only 
5 in  the  worse  case.  The  correlation  values  in  the  threshold  range  65 
to  95  were  found  at  approximately  the  same  coordinate  locations,  with 
only  a two-column  shift  occurring  in  some  cases. 

Due  to  the  nature  of  the  curve  shown  in  Figure  39,  an  iterative 
search  technique  to  locate  the  global  maximum  would  be  time  consuming. 
Since  the  correlation  peak  is  relatively  insensitive  to  IRIS  threshold 
within  a wide  threshold  range,  it  is  probably  not  necessary  to  find  an 
exact  optimum  threshold  value.  To  insure  valid  correlation  results  it 
may  only  be  necessary  to  identify  a small  threshold  range  which  contains 
the  optimum  value.  This  could  be  done  by  adjusting  the  IR  threshold  in 
successive  fields  (possibly  3 or  4)  and  selecting  the  one  which  gives 
the  peak  correlation  value. 

A second  method  of  IR  gradient  threshold  selection  was  developed  for 
use  in  the  simulation  program.  The  method  is  an  efficient  way  to  generate 
the  data  for  curves  such  as  the  one  shown  in  Figure  39.  The  method  oper- 
ates in  the  following  manner.  Suppose  that  a K X L binary  reference  has 
been  selected  and  that  the  K X L IR  gradient  array  has  been  generated 
which  will  best  fit  the  reference  with  proper  threshold  selection.  The 
first  step  in  the  process  is  to  sort  the  IR  gradient  into  two  vector 
arrays  depending  on  the  binary  value  of  the  corresponding  pixel  in  the 
reference  array.  As  an  example  let  X(i,  j)  be  the  reference  array  value 
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of  pixel  (i,  j).  Let  Y(i,  j)  be  the  IR  gradient  value  of  pixel  (i,j). 

Then 

YONE  (I)  = Y(i , j)  if  X(i,  j)  = 1 and 

YZERO  (I)  = Y(i,  j)  if  X(i,  j)  = 0 

where  YONE(-)  and  YZERO ( • ) are  vector  arrays  of  appropriate  IR  gradient 
values.  The  next  step  is  to  place  YONE(-)  and  YZERO(-)  in  ascending 
order  of  magnitude.  The  minimum  and  maximum  gradient  values  among  the 
two  vector  arrays  are  identified  as  limits  for  the  range  of  gradient 
threshold.  Starting  with  the  minimum  gradient  value  the  threshold  is 
now  varied  throughout  its  range.  To  determine  the  resulting  correlation 
estimate  for  a given  threshold  value,  the  Y0NE(*)  and  YZERO ( * ) arrays 
are  searched.  Those  values  equal  to  and  above  the  threshold  in  the 
Y0NE(*)  array  would  yield  1 - 1 (Ref  to  IR)  matches  and  those  values 
below  the  threshold  in  the  YZER0(*)  array  would  yield  0-0  matches. 
Therefore,  the  process  of  estimating  the  correlation  values  as  the 
threshold  is  varied  is  one  of  keeping  track  of  the  number  of  1 - 1 and 
0-0  matches  in  the  Y0NE(*)  and  YZER0(*)  arrays.  Figure  40  describes 
the  process  graphically,  where  YONE(-)  and  YZER0(*)  have  been  ordered. 

The  process  continues  by  setting  the  threshold  at  the  next  highest 
gradient  value  among  the  two  arrays.  Once  all  threshold  values  have  been 
set,  the  threshold  value  yielding  the  highest  correlation  value  is  selected 
as  the  optimum.  The  optimum  threshold  value  insures  the  best  fit  of  the 
IR  binary  array  to  the  given  binary  reference  array.  This  method,  while 
not  practical  for  real  time  implementation,  has  been  of  great  benefit  for 
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simulation  purposes  in  getting  a quick  idea  of  where  to  set  the  IR 
gradient  threshold  for  correlation  simulations. 

F.  Difficulties  Encountered  in  the  Simulation  Process 
During  the  course  of  running  the  simulations  reported  on  here,  a 
number  of  difficulties  were  encountered.  This  section  lists  those  diffi- 
culties and  explains  the  methods  by  which  they  were  rectified. 

1 . Scale  Factor  determination  between  HR-TV  and  Sensor  Images 

In  the  first  simulations  attempted  for  TV-to-TV  and  TV-to-IR 
correlations  is  was  suspected  that  small  errors  in  the  horizontal  and 
vertical  scale  factors  for  HR-TV  resolution  reduction  was  affecting  the 
goodness  of  the  correlation  function.  Therefore,  it  was  decided  that  a 
scale  factor  sensitivity  analysis  be  made  for  these  two  types  of  correla- 
tions. The  results  of  those  studies  will  now  be  presented. 
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a.  TV  to  TV  Correlation  Scale  Factor  Sensitivity 
A number  of  simulation  runs  were  made  to  determine  the  optimal  scale 
factor  reduction  for  high  resolution  TV  to  low  resolution  TV  correlation. 
The  simulations  used  a line  average  quantizer  which  set  each  pixel  to  0 
or  1 based  on  whether  its  value  was  below  or  above  the  average  value  of 
the  previous  eleven  pixels.  The  reference  image  in  each  case  was  31  by 
31  pixels  which  would  limit  the  maximum  correlation  surface  value  to  31 
squared  or  961 . 

Table  6 gives  the  results  obtained  using  scene  1 which  is  the  NASA 
building  with  a crane  and  antenna  on  its  roof.  From  the  table  it  is 


obvious  that  the  optimum  scale  factor  for  the  HR-TV  to  LR-TV  correlation 
is  4 to  1 in  both  axes.  Although  Table  6 does  not  show  any  results  for 
different  scale  factors  in  the  two  axes  (e.g. , 3.8  in  vertical  axis  and 
4.0  in  horizontal  axis),  simulations  were  run  with  the  result  that  the 


Scale  Factor 

Vertical  X Horizontal 

RMAX 

I MAX 

JMAX 

3.6  x 3.6 

639 

33 

21 

3.8  x 3.8 

639 

33 

21 

4.0  x 4.0 

644 

30 

25 

4.2  x 4.2 

| 

638 

34 

20 

4.4  x 4.4 

634 

32 

J 

23 

Table  6.  Scale  factor  sensitivity  analysis  - Scene  1 


4.0  scale  factor  reduction  in  each  axis  is  optimal.  The  variation  in 
the  values  of  the  row  and  column  at  which  the  peak  occurs  is  caused  by 
the  fact  that  as  the  scale  factor  is  changed,  the  location  of  the  chosen 


reference  within  the  FOv  changes.  The  peak  in  each  case  was  well  defined 
and  located  at  the  predicted  value. 

Table  7 gives  the  results  of  the  same  analysis  using  scene  3 which 
is  a wooded  area.  Although  the  peak  at  a 4.0  by  4.0  reduction  factor  was 
low,  it  is  felt  that  this  was  caused  by  a slight  shift  in  the  reference. 
If  one  considers  all  of  the  data  points  in  Table  7,  the  optimal  value 
appears  to  be  4.0  by  4.0  which  was  the  expected  value  from  the  HR  and  LR 
TV  specifications.  The  sharpness  of  the  peaks  (an  indication  of  the 
quality  of  the  peak)  was  best  for  the  3.8  x 3.8  and  4.0  x 4.0  scale 
factors. 


Horizontal  Scale  Factor 


3.6 

3.6  RMAX  = 593 


618  RMAX  = 601 


RMAX  = 605 

RMAX  = 609  RMAX  =618 


RMAX  = 606 


Table  7.  Scale  factor  sensitivity  analysis  - Scene  3 
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b.  TV-to-IR  Correlation  Scale  Factor  Sensitivity 
A number  of  simulation  runs  were  made  to  determine  the  optimal  scale 
factor  reduction  for  high  resolution  TV  to  IR  correlation.  The  simula- 
tions used  the  2x2  Roberts  Cross  Operator  with  preset  thresholds  to 
create  the  binary  images.  Two  sets  of  simulations  were  run.  The  first 
used  a reference  size  of  32  x 20  which  limits  the  maximum  correlation 
surface  value  to  640  and  the  second  set  used  a reference  size  of  16  x 20 
which  would  limit  the  maximum  correlation  surface  value  to  320.  The 
16  x 20  reference  was  also  used  to  remove  the  wide  side  peaks  as  dis- 
cussed in  section  E of  this  chapter. 

Tables  8 and  9 give  the  results  obtained  using  scene  1 which  is  the 

v 

NASA  building  with  a crane  and  antenna  on  its  roof.  Although  the  peak 


Horizontal 

Scale  Factor 

RMAX 

16 

505 

17 

505 

18 

511 

19 

511 

20 

505 

Table  8.  Horizontal  scale  factor  sensitivity  analysis 
with  vertical  scale  factor  equal  to  4.722  - 
Scene  1 


at  a 4.722  x 18.0  (Vertical  x Horizontal)  reduction  factor  is  lower  than 
some  of  the  others  it  is  felt  that  these  are  the  optimum  scale  factors 
when  one  considers  sharpness  of  the  peaks  and  the  ratio  of  peak  values  to 
the  false  peaks  in  both  axis  cross-sections. 
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Vertical 

Scale 

Factor 

RMAX 

I MAX 

JMAX 

3.5 

28 

57 

4.0 

29 

57 

4.5 

. 

517 

30 

57 

4.722 

511 

29 

57 

5.0 

520 

29 

57 

5.5 

524 

29 

57 

Table  9.  Vertical  scale  factor  sensitivty  analysis 
with  horizontal  scale  factor  equal  to  18.0 
Scene  1 


The  results  using  a 16  x 20  reference  supported  the  conclusions 

made  above.  Only  the  vertical  scale  factor  was  changed  in  these  runs. 

Again  the  results  for  a vertical  scale  factor  of  4.722  was  optimum. 

2 . Sampling  Rate  for  the  IRIS  Video  Due  to  the  Dynamic  Response  of  the 
IR  Sensors 


Each  field  of  video  consists  of  240  lines  with  each  line  sampled  to 
512  pixels  yielding  a 240  x 512  array  for  each  field.  Each  line  consisted 
of  all  of  the  active  video  and  a small  part  of  the  horizontal  retrace. 
Since  the  field  rate  is  60  Hz,  the  video  sampling  rate  is  approximately 
10  MHz. 

When  the  edge  detection  algorithms  were  first  applied  to  the  IRIS 
video,  it  was  noticed  that  very  few  vertical  edges  were  present  in  scenes 
that  were  known  to  have  vertical  edges  such  as  the  NASA  building.  The 
reason  for  this  is  that  the  rate  of  change  of  the  video  along  a line  is 
dependent  on  the  instantaneous  horizontal  field  of  view  of  the  IRIS 
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detectors  and  their  dynamic  response.  In  other  words  by  oversampling 
and  using  the  Robert's  two-by-two  edge  detection  algorithm,  vertical 
edges  were  not  being  detected. 

This  phenomena  can  be  explained  by  considering  the  characteristics 
of  the  IR  detectors.  The  response  time  of  the  IR  detector  is  420  ns. 

If  the  detector  is  modeled  as  a first  order  lag,  its  transfer  function 
is  given  by 

Vout  1 _ 1 _ 1 

V-n  " TS+1  . 42s+l  l+s/2.38 

where  the  time  constant,  T,  is  in  microseconds. 

An  edge  in  the  video  would  appear  as  a step  input  signal  (A/s). 

The  rise  time  of  the  detectors,  however,  determine  the  speed  of  the 
response.  Figure  41  shows  that  with  a detector  time  constant  of  420  ns, 
sampling  the  video  at  approximately  10  MHz  yields  approximately  four 
samples  per  time  constant.  This  illustrates  why  even  an  abrupt  edge  is 
not  very  predominant  in  an  edge  detection  algorithm. 


Figure  41.  IR  detector  response  to  an  edge. 


The  above  results  led  to  the  conclusion  that  even  though  the  original 
video  was  sampled  at  approximately  10  MHz,  only  every  fourth  sample 
should  be  used  thereby  reducing  the  effective  sampling  rate  to  c'.pproxi- 
mately  2.5  MHz.  Simulations  showed  that  correlation  was  impossible  using 


every  sample  (10  MHz  sampling  rate)  and  not  very  good  using  every  other 
sample  (5  MHz  sample  rate)  but  was  good  using  every  fourth  sample  (2.5 
MHz  sample  rate). 
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4.  SENSORS  AND  STATISTICAL  MODEL 

In  this  section  two  subjects  are  treated.  First  a descriptive  review 
of  imaging  sensor  characteristics  is  given  in  terms  of  typical  performance 
parameters  and  their  inter-relationships.  Secondly,  further  analysis  of 
threshold  statistics  for  the  binary  correlator  is  given  which  seeks  to  place 
the  analysis  on  a more  sound  theoretical  base  and  to  include  the  very  real 
possibility  of  pixel  mis-match  at  registration  due  to  sensor  contrast 
reversals  or  geometric  distortions.  A continuation  of  previous  work  is 
presented  which  allows  inclusion  of  pixel  mis-match. 

An  area  which  is  discussed  only  qualitatively  is  the  effects  expected 
when  all  pixels  in  an  image  cannot  be  considered  independent.  A quantita- 
tive treatment  of  these  effects  is  strongly  dependent  on  specific  scene 
characteristics  and  was  not  done  for  the  present  effort.  A quantitative 
analysis  based  on  judiciously  selected  scene  characteristics  should  be 
considered  for  further  work. 

In  particular,  the  false  alarm  problem  is  typically  analyzed  based  on 
an  assumption  of  totally  uncorrelated  images.  This  gives  the  familiar  error 
function  type  false  alarm  curve  which  is  highly  optimistic.  A more  realistic 
analysis  is  needed  which  includes  the  effects  of  spatial  frequency  content 
of  the  images  and  allows  for  practical  cases  where  the  correlation  function 
has  an  inherent  broad  peak. 

Else  a fast  algorithm  for  the  phase  correlation  method  should  be  sought. 
It  has  a much  sharper  inherent  correlation  peak. 
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A.  Characteristics  of  Passive,  Infrared, 

Imaging  Sensors 

Performance  characteristics  for  any  electro-optical  sensor,  be  it 
passive,  active,  imaging  or  non-imaging,  are  typically  described  in  terms 
of  a few  selected  parameters.  This  set  of  parameters,  while  not  unique, 
typically  includes  a measure  of  signal -to-noise  ratio,  spatial  resolution 
(or  MTF),  time  response,  spectral  band,  dynamic  range,  noise  equivalent 
flux  density  (NEFD),  etc.  Additionally  other  related  parameters  or 
application  specific  parameters  may  be  defined  for  a given  system. 

Although  some  understanding  of  the  effect  of  these  parameters  is 
achieved  from  their  definitions,  a more  complete  understanding  requires  a 
detailed  parametric  analysis.  This  is  complicated  by  the  fact  that  many 
parameters  exhibit  a complex  interactive  dependence  on  more  than  one  element 
of  the  total  system.  Additionally,  it  is  necessary  to  include  parameters 
which  exhibit  both  radiometric  and  electrical  properties.  The  necessary 
conversions  from  radiometric  to  electrical  quantities  are  typically  given 
in  terms  of  responsivity  of  the  photodetector.  Also,  the  system  noise  in- 
cludes several  possible  sources  dependent  on  the  specific  sensor  design. 

Rather  than  bog  down  in  the  detailed  analysis  of  a specific  sensor 
design,  the  following  discussion  will  be  limited  to  fundamental  concepts 
which  have  general  application.  These  concepts  are  best  discussed  in  terms 
of  the  following  sub-categories. 

•Scene  characteristics 

•Geometry  effects 

•Optical  path 

•Ambient  radiation  characteristics 
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•Optics  (lenses,  mirrors,  optical  filters,  baffle  design) 

•Photodetector 

•Imaging  mechanism 

•Electronics 

The  concepts  are  discussed  in  terms  which  hopefully  convey  physical 
understanding  instead  of  terms  which  give  detailed  parametric  equations. 

The  latter  can  be  provided  but  would  require  in-depth  study  for  understanding. 

1 . Scene  Characteristics 

Except  for  certain  coherent  or  selective  radiators,  the  radiation  from 
a typical  scene  consists  of  two  components:  1.  emitted  thermal  radiation 
dependent  on  scene  emissivity  and  temperature  profiles,  and  2.  reflected 
radiation  dependent  on  the  scene  reflectivity  and  its  ambient  radiation 
environment  (typically  sunlight).  Normally  for  wavelengths  of  interest,  the 
viewed  scene  can  be  assumed  to  be  opaque  (transmissivity  = 0).  The  resultant 
combined  radiation  characteristics  provide  a spectral  signature  for  a given 
scene.  This  spectral  signature  is  one  of  the  major  inputs  to  a sensor 
design  analysis. 

Spectral  signatures  can  be  determined  in  a number  of  ways  including: 

Experimental  - fortunately  data  are  available  for  radiation  properties 
of  a large  number  of  materials  and  specific  target  compositions  (experimentally 
measured).  Unfortunately  these  data  are  often  scattered  through  various 
references  and,  more  importantly,  often  measured  under  conditions  which  do 
not  match  the  intended  application  scenario. 

Theoretical  - requires  assumptions  (basically  simplifying)  about  the 


A 


spectral  dependence  of  target  emissivity,  reflectivity  and  ambient 
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environment  as  well  as  about  expected  target  temperature  and  cross-section. 

The  advantage  of  this  technique  is  that  ball  park  figures  are  easily 
obtained  and  modified. 

Hybrid  - basically  employs  the  simplicity  of  the  theoretical  approach 
but  uses  experimental  data  for  the  more  common  and  easily  found  radiation 
properties. 

2.  Geometry  Effects 

Most  objects  within  a scene  can  be  assumed  to  be  diffuse  emitters/ 
reflectors  with  isotropic  radiation  properties.  This  assumption  is  usually 
made  in  propagating  the  spectral  signature  from  the  scene  to  the  sensor 
(results  in  a 1/R  reduction  in  radiance  level).  Other  geometry  effects 
have  to  do  with  the  size  and  signal  strength  of  an  object  within  the  field- 
of-view  projected  onto  the  photosensor.  For  example  a target  subtends  a 
total  solid  angle  at  the  sensor  given  by  the  ratio  of  target  area  to  range 
squared.  This  information  is  useful  in  determining  the  percentage  of  the 
total  sensor  field-of-view  occupied  by  the  target.  Further,  signal  strength 
on  the  photodetector  increases  as  the  primary  optical  area  increases.  That 
is,  the  collecting  solid  angle  of  the  primary  optics  for  the  sensor  is 
given  by  the  ratio  of  optics  area  to  range  squared. 

3.  Optical  Path 

For  free  space  propagation  the  target  characteristics  and  geometry 
effects  are  sufficient  to  determine  the  radiant  signal  levels  on  the  photo- 
surface. However,  for  propagation  through  the  atmosphere  or  any  other 
refracting  medium  consideration  must  be  given  to  1.  radiance  reduction 
through  absorption  or  scattering  out  of  the  sensor  field-of-view,  2.  radiance 
increases  through  scattering  of  background  radiance  into  the  sensor 
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field-of-view,  and  3.  wavefront  distortion  due  to  turbulence  effects. 

Effects  1.  and  2.  tend  to  reduce  the  contrast  within  the  image.  Effect 

3.  causes  a blurring  effect  whose  time  averaged  result  is  a reduction  in 
resolution  of  the  image.  The  first  two  affect  signal  level;  the  third 
affects  MTF. 

4.  Ambii-nt  Radiation  Characteristics 

For  typical  imaging  applications,  the  ambient  radiation  environment 
includes  incident  and  reflected  sunshine,  incident  and  reflected  earthshine 
(significant  at  longer  wavelengths),  earth  limb  or  airglow  and  possible 
high  radiance  sources  not  in  the  field-of-view.  Depending  cn  the  spectral 
region  of  interest  and  the  specific  application,  various  elements  of  the 
ambient  radiation  environment  represent  either  signal  or  noise  to  the 
sensor.  Ideally,  the  sensor  "sees"  only  the  intended  target  area  through 
a perfectly  clear  optical  path  with  no  extraneous  radiation  entering  the 
sensor.  Methods  for  improving  the  sensor's  ability  to  accomplish  the 
ideal  objective  include  judicious  selection  of  spectral  band,  optical 
filtering  to  exclude  wavelengths  of  no  interest,  optical  design  and  baffle 
design  to  essentially  trap  unwanted  stray  radiation.  Even  with  the  best 
application  of  these  methods  there  will  be  some  stray  radiation  in  the 
detector  field-of-view  which  cannot  be  eliminated. 

5.  Optics 

As  mentioned  above  the  optical  design  has  as  one  goal , the  optimum 
collection  of  scene  radiation  and  rejection  of  stray  radiation.  Additionally, 
the  optics  must  provide  image  magnification  where  needed,  minimuze  distortion 
and  aberrations  from  the  optical  elements,  and  provide  transmission  char- 
acteristics consistent  with  the  desired  spectral  band  of  sensitivity.  The 
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optics  in  conjunction  with  detector  size,  target  area  size  and  sensor-to- 
target  range  determine  the  sensor  field-of-view  and  basic  resolution. 

6.  Photodetector 

Photodetectors  are  a class  of  optical  detectors,  consisting  of  several 
different  types,  whose  electrical  properties  are  modified  through  the 
absorption  of  discrete  quanta  of  optical  energy,  or  photons.  This  absorp- 
tion results  in  ionizations  which  produce  charge  carriers.  In  contrast, 
thermal  detectors  absorb  optical  radiation  resulting  in  increased  lattice 
vibration.  These  lattice  vibrations  show  up  in  temperature  dependence  of 
the  detector  electrical  properties. 

The  photodetector  is  the  heart  of  any  sensor  and  effectivity  deter- 
mines the  signal -to-noise  ratio,  time  response  and  dynamic  range  of  the 
sensor.  The  photodetector  is  characterized  by  several  parameters  and 
functions  including  responsivity,  time  constant,  detectivity  and  size. 

Responsivity  is  a measure  of  the  detector's  ability  to  convert  optical 
radiation  into  an  electrical  signal.  It  typically  has  units  of  volts/watt 
or  amps/watt.  Responsivity  is  dependent  on  wavelength,  detector  operating 
temperature  and  frequency.  Further,  the  dependence  on  these  three  para- 
meters is  inter-related.  Obviously,  in  choosing  a detector  for  use  in  a 
sensor  design,  the  detector  should  have  high  responsivity  at  the  dominant 
wavelengths  in  the  scene  spectral  signature. 

Time  constant  for  a photodetector  is  dependent  on  the  mean  lifetime  for 
photo-generated  charge  carriers.  This  carrier  lifetime  is  a complex  func- 
tion of  detector  type,  dopant  material,  physical  size  and  detector  temperature. 
These  characteristics  as  well  as  responsivity  and  other  photodetector 
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performance  curves  must  be  provided  by  the  manufacturer.  An  additional 
factor  which  affects  time  response  and  detectivity  is  the  level  of  total 
irradiance  on  the  detector,  including  both  scene  and  stray  radiation  plus 
any  thermal  radiation  from  the  sensor  optical  elements.  A reasonably 
simple  model  can  be  devised  to  show  how  time  constant  depends  on  the 
above  parameters. 

Detectivity  is  a measure  of  how  sensitive  a photodetector  is;  a large 
detectivity  is  good.  An  equivalent  performance  measure  often  used,  which 
is  just  the  reciprocal  of  detectivity,  is  NEP  or  noise  equivalent  power 
(measured  in  watts).  The  NEP  gives  the  incident  radiant  power  level  in 
watts  which  will  produce  a unity  signal-to-noise  ratio  out  of  the  detector. 
The  output  signal  and  noise  levels  depend  upon  the  detector  area  and  the 
electronic  bandwidth  of  the  detector  circuitry.  Thus  NEP  is  not  a very 
good  comparison  parameter  unless  the  detector  size  and  bandwidth  effects 
are  accounted  for.  This  is  done  by  defining  a specific  detectivity,  called 
D*,  which  is  normalized  to  a detector  area  of  one  square  centimeter  and  a 
noise  bandwidth  of  one  Hertz.  Curves  of  D*  versus  wavelength  are  often 
used  to  compare  various  detectors.  Further,  D*  depends  on  total  incident 
radiation  level  frequency  and  exhibits  a threshold  dependence  on  detector 
temperature. 

A performance  measure  which  characterizes  the  entire  sensor  including 

all  elements  from  the  optics  through  the  electronics  is  the  sensor  NEFD  or 

noise  equivalent  flux  density.  NEFD  is  the  level  of  incident  radiant  flux 
o 

(in  watts/m  ) on  the  sensor  which  produces  a signal-to-noise  of  unity.  Thus 
it  is  similar  to  NEP  except  for  the  units  and  is  easily  related  to  sensor 
signal-to-noise  ratio.  For  example  we  have  S/N  = E/NEFD  where  E is  the 
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scene  produced  irradiance  level  at  the  sensor  (depends  only  on  the  scene 
and  path  characteristics)  and  NEFD  is  dependent  only  on  sensor  parameters. 
Analysis  of  how  various  sensor  parameters  affect  NEFD  is  a useful  ''esign 
tool . 

The  effect  of  detector  size,  in  addition  to  those  effects  already 
discussed,  is  to  limit  the  system  spatial  resolution.  As  a rough  first 
estimate,  the  limiting  resolution  can  be  taken  to  be  the  instantaneous 
field-of-view  in  each  direction  subtended  by  the  detector.  This  is  the 
ratio  of  detector  width  or  length  to  the  effective  focal  length  of  the 
optical  system.  Some  enhancement  of  this  spatial  resolution  can  be 
achieved  electronically.  The  electronics  is  then  said  to  provide  an  MTF 
boost  to  the  overall  system  resolution.  For  total  system  resolution  it  is 
usually  convenient  to  represent  each  element  of  the  system  by  an  appropriate 
MTF  or  modulation  transfer  function  and  take  the  products  to  get  the  total. 
7.  Imaging  Mechanism 

For  point  detectors  imaging  is  accomplished  by  scanning  the  detector 
field-of-view  across  the  scene  to  be  imaged.  This  scanning  is  typically 
accomplished  mechanically,  either  with  rotating  or  mutating  mirrors  or 
with  gimbals.  Requirements  on  the  mechanical  scanning  can  be  relaxed  by 
using  either  linear  or  two-dimensional  arrays  of  detector  elements.  CCD 
and  CID  arrays  of  photodetectors  are  now  available  for  a few  wavelength 
regions.  Of  course,  the  price  paid  for  using  arrays  of  photodetectors  is 
increased  electronics  complexity  plus  the  possibility  that  some  elements  in 
the  array  will  be  defective. 

Area  detectors  such  as  used  in  Vidicons  and  other  TV  type  sensors 
achieve  scanning  and  readout  with  an  electron  beam.  This  has  advantage  of 
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simplicity  but  is  less  rugged  than  solid  state  photosensors. 

8.  Electronics 

The  electronics  must  drive  the  scan  mechanism,  amplify  and  process  the 
signal.  For  longer  wavelength  infrared  systems,  the  first  amplifier  stage 
is  also  a significant  source  of  noise.  Typically,  both  the  detector  and 
preamplifier  are  cooled  to  reduce  thermal  noise.  For  systems  operating  in 
the  visible  region,  noise  is  much  less  of  a problem  because  of  the  availabil- 
ity of  higher  D*  detectors  and  because  fewer  thermally  generated  carriers 
will  exist  in  a detector  whose  energy  bandgaps  represent  visible  wavelengths. 
Thus  cooling  is  not  usually  required  for  visible  wavelengths. 

As  already  mentioned,  the  electronics  can  boost  system  resolution  and 
can  be  used  to  further  process  the  signal  for  utilization. 

In  summary,  the  above  descriptive  discussion  has  introduced  many  of 
the  primary  parameters  affecting  a sensor  design  or  analysis.  Dynamic 
range  is  determined  by  the  photodetector  characteristics  and  is  the  range 
of  linear  responsivity.  The  lower  end  is  limited  by  the  detector  NEP  and 
the  upper  end  is  limited  by  the  irradiance  level  at  which  the  detector 
saturates.  Saturation  occurs  when  no  further  charge  carriers  can  be  gener- 
ated and  is  a function  of  the  detector  materials,  doping  levels  and  other 
detector  characteristics.  Dynamic  range  can  be  extended  by  the  sensor  in 
steps  by  optical  attenuation  of  the  incoming  radiation.  The  lowest  end  is, 
however,  still  that  determined  by  NEP. 

B.  Statistical  Model  for  Threshold  Analysis 

An  analysis  for  determination  of  probability  of  correct  registration 
(Pj)  and  probability  of  false  registration  (Pp)  as  functions  of  choice  of 
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threshold,  image  signal -to-noise  ratios  and  image  statistics  was  done  for 
correlation  of  two  binary  quantized  TV  images.  Results  of  this  analysis 
are  given  in  [4]  and  further  developed  in  [6].  This  analysis  was  based  on 
two  simplifying  assumptions:  1.  The  images  experienced  no  geometric 
distortion  and  2.  all  the  pixels  across  an  image  were  statistically  in- 
dependent. Further,  since  the  correlation  was  done  for  like  images  (both 
from  TV  visible  spectrum  sensors)  it  was  assumed  that  at  registration,  the 
images  were  identical  except  for  additive  noise  terms. 

In  the  context  of  the  present  task  of  correlating  TV  with  FUR  images 
the  assumption  of  identical  images  at  registration  can  no  longer  be  made. 
Additionally,  there  is  greater  likelihood  that  geometric  distortion  will 
be  present.  These  modifications  to  the  statistical  model  were  examined  for 
a binary  TV  to  FLIR  correlator. 

In  addition,  a re-evaluation  of  the  statistical  properties  for  a 
binary  correlator  were  examined  in  terms  of  "measure  of  goodness"  for  esti- 
mates of  statistical  functions.  The  correlation  estimate  was  found  to  be 
both  unbiased  and  consistent  with  a variance  proportional  to  one  over  the 
number  of  data  points  (KL)  for  the  distortionless  case.  It  was  further 
verified  how  the  estimate  variance  is  scene  dependent. 

1 . Statistical  Analysis  Theory 

In  re-evaluating  a statistical  model  for  describing  correlator  perfor- 
mance for  TV  to  FLIR  correlation,  a re-examination  has  been  made  of  the 
goodness  of  the  direct  correlation  estimate  for  the  cross-correlation 
function. 


If  we  define  s^x.y)  to  be  the  two-dimensional  time-frozen  image  from 
the  reference  sensor  and  S2(x,y)  to  be  the  two-dimensional  time-frozen 
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image  from  the  missile  sensor  then  the  cross-correlation  function  between 
s^  and  S2  is  given  by 

R12  <V  V = E£S1<V  V s2(xo  + «»  y0  + 

where  E[  ] implies  "expected  value  of"  and  since  typical  imagery  is 
known  to  be  shift-variant,  depends  not  only  on  the  relative  offset 
(a,  b)  between  s^  and  s2  but  on  the  coordinate  origin  (xQ,  yQ)  also.  Now 
if  we  have  only  one  member  function  for  each  of  s-j  and  s2,  without  knowing 
the  joint  probability  density  function  for  s^  and  s2,  R^2  cannot  com- 
puted  exactly.  Thus,  we  estimate  the  value  of  R12  based  on  K by  L samples 
from  s-j  and  M by  N samples  from  s2  to  give 

L K 

R12^xo’  yo’  p’  = ~J£  X/S  Mn*  s2^n  + P»  m + O9) 

n=l  m=l  | o-p-N-L 

for  J < < 

( o-q-M-K 

where  R^2  is  an  estimate  for  R12 
x = nAx  a = pax 

(20) 

y = mAy  e = qAy 

Ax,  Ay  are  x and  y sample  spacing. 

A 

If  R.j2  is  to  be  a good  estimate  for  R^2  it  must  be  unbiased  and  con- 
sistent. 

Unbiased:  E[R^2]  = R^2.  R^2  is  easily  shown  to  satisfy  this  requirement 

and  is  thus  unbiased. 

Consistent:  lim  E[(R12  - R,JZ]=  0.  (21) 

KL-**' 

A 2 

where  the  term  E[(Rj2  - R12)  ] is  simply  the  variance  of  the  estimate 
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about  the  true  value  and  for  a consistent  estimate,  R^>  the  variance 
goes  to  zero  as  the  number  of  data  values  used  in  estimating  R12  yets 
large.  Further,  the  estimate  variance  is  a useful  parameter  in  evaluating 
the  confidence  to  be  placed  in  a single  computation  of  the  cross-correla- 

A 

tion  R^2*  In  other  words  it  is  relatable  to  the  probability  that  a given 
R^2  is  within  a given  error  of  the  true  R^- 

For  the  estimate  R^»  as  defined,  consistency  is  proven  for  noise- 
free,  distortion-free  case  below 

VAR[R12]  = E[R12  - R12)2]  = E[R122]  - R22  (22) 

L K L K 

where:  ECR^2]  = E[(^|-)2  s1(n,m)s2(n+p,  m+q) 

n=l  m=l  i=l  j=l 

•Sj (i , j)s2(i+P,  j+q)] 

or  using  the  linearity  of  E [ ], 

L K L K 

E[R]22]  = 22Xrf2E^sl^n’  m)  s2(n+p’  m+q)  sl^’  j) 
n=l  m=l  i=l  j=l 

•s2(i+P,  j+q)] 

Then  from  [7] 

E[s1  s2s3s4]  = E[s-, s2]  E[s3s4]  + ECs3 s3]  E[s2s4]  (23) 

+ E[Sls4]  E[s2s3]  - 2uslus2Ps3ys4 

and  assuming  the  mean  values  have  been  removed,  us.j=0,  all  i,  we  have 

L K L K 

e[r12  ] = (-j^)2  R12  ^P’  + Rll^1'n’ 

n=l  m=l  i=l  j=l  (24) 

•R99(1-n,  j-m)  + R19(i+p-n,  j+q-m)R,, (i-p-n,  j-q-m) 
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where  and  R 22  are  the  autocorrelation  functions  for  Sj  and  s2 
respectively  and  R,^  and  R^  are  cross  correlation  functions  for  s-j  and 


2 

The  R^  terms  obtained  are  exact  functions;  further  R^  (p.  q)  is 
independent  of  the  sumnation  indices.  Using  this  result,  the  VAR[R^] 
becomes 

2 L-l  K-l 

%2  = VAR£Ri2(V  V p»  ^ = (^r)2  2 U--l*l)0Hfe|) 

£=-(L-l)  fe=-(K-l ) 

.{R11(£,  fe)R22(£,  fe)  + R12(£+p,  fe+q)R21U-p,  M)>  (25) 


L-l 


K-l 


R12  = ~RT  ^ ^ ^ ^ (1~  •^-)0~  -^-HRj  ■)(•£»  fe)^22^*  ^) 

£=-(L-l ) fe=-(K-l ) (26) 

+R]2U+p,  fe+q)R21(£-p,  fe-<?) } 


where  the  following  changes  of  variables  have  been  made 

l=i-n  ; -(L-l)  < l < (L-l) 
k=j-m  ; -(K-l)  < fe  < (K-l) 


(27) 


and  the  L-  |£|,  K-  (fe|  terms  arise  because  for  each  value  of  l 6 fe  there 
are  (L-|£| ) ( K- | fe j ) identical  terms  under  the  quad  sum. 

If 


EE  {R^U,  fc)R22U,  fe)  + R12U+p,  fe+<?)R21U-p,  k-q) 

Ik 

2 * 

for  all  x , y , p and  q then  clearly  lim  o„  =0  and  R,?  is  a 
00  KL^»  R12  lc 

consistent  estimate  of  the  true  cross  correlation  function  R12. 
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Of  further  significance  is  that  o„  shows  how  the  variance 

R12 

(and  thus  reliability,  confidence  interval,  error  probability)  depends 
on  not  only  the  coordinate  origin  (xq,  y ) but  on  individual  shift 
values  within  a given  correlation  surface  estimate.  Dependence  on  scene 
statistics  is  inherent  in  the  Rn  & r22  terms  in  the  variance. 

The  above  development  formalizes  the  statistical  approach  to  error 
probability  and  registration  probability  for  the  noise-free  distortion-free 

If 

cross-correlation  function  estimate.  Noise  is  easily  included  by  replac- 
ing s-j  and  s2  .with  s.j  + n-j  and  s2  + n2  respectively.  Similar  results  are 

l *■ 

expected. 

2.  Correct  Registration 

The  following  development  is  an  extension  of  the  work  in  [7]  to  include 
the  situation  where,  at  registration,  there  are  pixel  mis-matches  between 
the  two  images  (in  the  absence  of  noise).  These  pixel  mis-matches  may  be 
caused  by  contrast  reversals  between  the  TV  and  FUR  sensors  or  may  be 
caused  by  image  distortion.  The  effect  of  these  mis-matches  is  incorpora- 
ted into  the  model  in  [7]  by  the  parameter,  e,  which  is  defined  to  be  the 
probability  that  a FUR  and  TV  pixel  do  not  match,  given  that  the  images 
are  registered.  There  is,  unfortunately,  still  a problem  in  determining 
a realistic  value  for  c since  it  will  be  scene  dependent. 

The  assumption  that  all  pixels  are  independent  as  used  in  [7]  is  still 
employed  in  this  analysis.  This  assumption  has  its  greatest  validity  for 
scenes  with  high  spatial  frequency  content  such  that  oversampling  is  not 
evident.  For  scenes  with  larger  objects  (lower  spatial  frequencies), 
considerable  oversampling  is  evident  and  the  assumption  of  independent 
pixels  begins  to  break  down.  The  result  is  a broader  peak  in  the 
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cross-correlation  function  and,  hence,  further  confusion  on  choice  of  a 
suitable  threshold  for  correlation  peak  detection. 

At  registration,  because  of  contrast  reversals  and  distortion,  the 
new  model  for  probability  of  correct  registration  has  the  following 
features: 

1.  Pixel  values  ^(n,  m)  and  U2(n+P0»  m+cl0)  are  not  necessarily 
equal  and  have  a probability  of  mismatch  = e. 

2.  Array  averages  and  T2  are  not  necessarily  equal. 

3.  The  pixel  mismatch  probability,  e,  is  exclusive  of  noise  induced 
mismatch. 

Thus,  for  the  n,  mth  pixel  pair  comparison  at  registration  we  have 

r-|  (n,  m)  = ^(n,  m)  + n^n.m) 

r2(m+po,  m+qQ)  = w2(n+p0>  m+qQ)  + n2(n+pQ,  m+qQ) 

where  the  noise  terms  n-j  and  n2  are  signal  fluctuations  due  to  both  optical 
and  electronic  causes  and  are  assumed  to  be  zero-mean  gaussian. 

Then  for  the  n,  mth  pixel  pair 

Pn  m [C]  = P[pixels  match/images  registered] 

= PCQt (r1 ) = 0 & g2(r2)  = 0]  (28) 

+ PCg-,  (r-, ) = 1 & g2(r2)  = 1] 

or  equivalently,  looking  at  all  possible  combinations  for  which  the  pixels 
match,  we  get 

Pn  m [C]  = P[r-j<T-j  & r2<^2^1<"^l  ^ lJ2<^2^  ^ y2<"^2^ 

(29) 

+ P [ r i < T-j  & r2<T 2 1 vi-j <T i & p2<T2]  P[ui<T^  & p2<T2] 


‘ 


104 

+P[rj>T1  & r2>T2lpl>^l  & U2>T2^  pCv* ^ >T -j  & y^>T2^ 

+P['r1>T-|  & r2>^2lpl<^l  & lJ2<^2^  ^ v*2<^2^ 

+P[r.|<T,  & r2<^2lpl<^l  ^ p2>^2^  ^1^1  ^ ll2>^2^ 

+P[^1  <T i & r^cT^ | p-j >T-j  & P2<^2^ 

+P[ r i >T i & ^^ly-^Ti  & pt|Ji>^i  & ^2*^2^ 

+P[ r*i >T -j  & i''2>T2|y-|<Ti  & ^ p2>^2^ 

Or  looking  at  the  domain  of  values  for  r-j  and  we  see  four  areas  delinea- 
ted by  lines  through  the  quantization  thresholds  T-j  & T^.  Then  in  terms 
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Pixels 

Match 


+ Pnl(ri"P]) 


SL 


+ 74l  Pnl^ri’Pl^  Pn2^r2'p2' 


•4L 

•H 


Pnl^rrwl^  Pn2^r2"w2^ 


Pnl ^rl _P1 ^ Pn2^r2"p2^ 


+ Pnl(rrlJl)  Pn2*r2"w2^ 

+ Pnl^rrpl^  Pn2^r2'p2^ 


• P[»"^ , r ^ in  A1  ] 
r^ , in  A1 

• P[r1 , r2  in  A3] 
r1 , r2  in  A3  1 

\ 

• P[^i » <"2  l n ^2] 
r1 , r2  in  A2 

• P[r^ » r2  in  A4] 

, r2  in  A4 

• P[r-j , r 2 in  A4] 

, r2  in  A4 

• P[rr  r2  ^n  A2] 
r-j , r2  in  A2 


y Pixels 
/Mismatch 


From  the  syirmetry  of  the  above  expression  and  recognizing  that 


P[r-|»  r2  in  A2]  + P[r.j,  r^  in  A4]  = e 
P[r-j,  r2  in  Al]  + P[r1 , r2  in  A3]  = 1 - e 


(31) 


further  if  we  define  signal -to-noise  ratios  for  the  two  signals  as 

* |pi(n,m)  - T,  | 

SN1  = — 1 


SN2  2 


°nl 

|p2(n+po,  m+qQ)  - T2| 
°n2 


(32) 


i.e.,  SN1  and  SN2  are  always  positive  and  held  at  constant  values  for 
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r^,  in  Al,  A2,  A3  or  A4,  then 

T T 


■V-  c°]  ■ 1 If  pni<rr“i) 


-oo  -00 


T T 


t(//  Pnl<rlV 


— OO  — oo 


oo  oo 

'//  Vvi’ 


T T 


oo  oo 

<//  pnl(rr1‘1) 


T T 
T T 


{ / / P.i(pr-i' 


T T 


<//  pnl(rr»1> 


— OO  — oo 


<//  pni‘’V-i> 


T T 


oo  oo 

+ {ff  pm(riV 


T T 


which  is  simplified  using  the  following. 

-(r-p)2 


J “ — - — e 2ac  dr 


and 


“ i 

/ /T7 


-(r-p) 

e 2o2  dr 


W^'WMHMH>ianillwal 


Pn2(r2"p2)} 

P[A3]: 

u i <T -j  ? p2<^2 

Pn2(r2"M2^} 

P[A_L] : 

y 1 >T] \ ^2>^2 

Pn2(r2"u2^} 

P[A1] : 

Pl>Tl;  p2>T2 

Pn2(r2_vl2^} 

P[A3] : 

U1<T1;  y2<T2 
(33) 

Pn2^r2_lJ2^ 

P[A2] : 

P^T-j  > P2>^2 

Pn2^r2"p2^} 

P[A4] : 

P-]  >l"l  > P£^2 

Pn2^r2"p2^} 

P[A4] : 

p i >1" i > ^2^2 

Pn2^r2_p2^} 

P[A2] : 

m1<T1:  m2>T2 

In  the  above,  we  need  to  evaluate 


(34) 


- 
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Making  a change  of  variable 


r-y  _ 
o 


da 


r ->•  T 


We  have 


vll 

/i 


-a 

2 

e da 


for  T > u 


-a 

2 

jr  1 e da 

J 

SN  2 

-a 

-SN  2“ 

r 1 e da 


f ■ — 


s\ 


2 

-a 

2~ 

e da 


for  w > T 


Then  we  get 


SN1  SN2 


Pn,m  M ' / / 1 * 


-SN1  SN2 


•wh  / / fc 

— oo  *00 

oo  oo 

pifl>]  / / h 


(«2  + 32; 


(«2  + B2: 


(a2  + B2] 


-SN1  -SN2 
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OD  00 

+ »[A3]  / / k e 
SN1  SN2 
SN1  -SN2 

* P[A2]  / / l e' 


(g2  + g2) 
2 


(g2  B‘ 


-SN1  SN2 


tp[A4]  //  k - 


— oo  — oo 


QO  OO 


/ / k 


-SN1  SN2 


oo  oo 


(g2  + B2) 
2 


(q2  + 62) 


(g2  + B2) 


+P[A2]  / / k 


SN1  SN2 

Now  making  use  of  the  following 

® -A  A <*> 

’ / * / 

-A  -®  -<*  A 


where 

then 


n 

y*  = j + erf  (A) 


A 1 


erf(A)  ' / e 


da 


P [C]  » P[A3]  {[4+erf(SNl)][4+erf(SN2)]  + 
n >«»» 

+ P[A1  ] { [4-erf  ( SNl ) ] [4-erf  ( SN2 ) ] + 
+ P[A2]  {[4+erf (SNl  )][4-erf (SN2)]  + 
+P  [A4]  ([4-erf (SNl )][4+erf (SN2)]  + 


or 

Pn  m [C]  = (1  - e)  (4  + 2 erf (SNl)  erf(SN2)} 

n fiii 

+ e (4  - 2 erf(SNl)  erf(SN2)} 


dud  3 

) 

1 

dads 

dad  8 

dadfi 

dadB 


(38) 


[4-erf (SNl )] [4-erf (SN2)]} 
[4+erf (SNl  )] [4+erf (SN2)]} 
[4-erf (SNl )] [4+erf (SN2)]} 
[4+erf (SNl )][4-erf (SN2)]> 

(39) 
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or  finally 


Pn  m [C]  ■ H + 2erf(SNl ) erf(SN2)  - 4eerf(SNl)  erf(SN2) 

II  9 III 


Pn,m  [C]  = **  + 2(l-e)[erf(SNl)  erf(SN2)] 


(40) 


This  result  is  interesting  in  that  we  have  derived  in  a cumbersome, 
brute-force  manner  what  should  be  intuitively  obvious.  That  is,  this 
result  differs  from  the  case  of  no  pixel  mis-matches  only  in  inclusion  of 
the  (1 -e ) term,  which  is  simply  the  probability  that  pixels  match  between 
the  TV  and  FLIR.  And  since  the  pixel  match  probability  and  the  probability 
of  choosing  pixels  matched  are  independent  of  each  other,  the  simple  multi- 
plication result  was  expected. 

It  is  emphasized  again  that  SN1  and  SN2  are  different  for  each  pixel 
pair  in  the  correlation  function  computation  so  that  for  KL  pixel  compari- 
sons one  can  talk  about  expected  values  for  Pn  m [C].  To  compute 
p = E[Pn  m [C]]  requires  knowledge  of  the  image  statistics  for  y-j  and  y^ 
as  well  as  knowledge  of  noise  variances. 

An  approach  similar  to  that  given  in  [7]  where  assumed  scene  statistics 
are  used  can  again  be  applied  and  carried  through  to  a detection  probability 
for  given  thresholds.  The  effect  of  e on  these  quantities  will  be  a general 
reduction  in  p and  thus  a reduction  in  Pp  for  any  given  threshold. 


5.  CONCLUSIONS  AND  RECOMMENDATIONS 


Most  of  the  conclusions  and  recommendations  resulting  from  the  work 

on  this  contract  have  been  given  and  justified  within  the  first  four 

chapters  of  this  report.  For  compactness  they  are  given  again  in  this 
chapter. 

1.  In  order  for  the  correlation  of  binary  images  to  be  a valid  measure 

of  similarity  it  is  necessary  that  the  quantization  process  be  based 

on  similar  measures  of  scene  content.  For  sensors  operating  in 
different  frequency  spectrums,  quantization  based  on  average  value 
or  standard  deviation  do  not  supply  similar  binary  images  to  the 
correlation  system.  Quantization  based  on  edge  content  overcomes 
this  limitation. 

2.  In  support  of  the  results  outlined  in  [4],  the  correlation  system 
simulations  performed  under  this  contract  show  that  TV-to-TV  correla- 
tions work  well  when  quantization  is  based  on  a line  average  or  array 
average  threshold.  The  simulations  indicate  that  the  above  threshold 
technique  which  yields  the  best  result  (sharpest  correlation  peak)  is 
scene  dependent. 

3.  From  the  set  of  simulations  conducted  for  TV-to-IR  correlations  using 
the  average  value  preprocessors  no  valid  correlations  were  found. 

It  is  unlikely  that  these  preprocessors,  acting  alone,  would  be  able 
to  provide  valid  and  reliable  correlations  when  the  sensors  are  non- 
compatible. 

4.  From  the  set  of  simulations  conducted  for  TV-to-IR  correlation  using 
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the  standard  deviation  preprocessors  no  valid  correlations  were 
found.  However,  there  are  factors,  not  studied  thoroughly  in  this 
effort,  which  could  effect  the  outcome  of  the  correlation  system. 

The  effects  of  factors  such  as:  1.  average  value  and  standard 
deviation  calculation  sample  size,  2.  proportionality  constant,  K, 
chosen  for  Ka  quantization  threshold,  and  3.  reference  image  size 
should  be  studied  more  thoroughly  before  these  preprocessors  are 
eliminated  for  TV-to-IR  correlation. 

5.  The  poor  performance  of  the  average  value  and  standard  deviation 
preprocessors  in  the  correlation  system  for  TV-to-IR  correlation 
led  to  the  use  of  the  edge  detector  preprocessors. 

6.  The  one/zero  distribution  in  the  reference  image  directly  effects 
the  quality  of  the  correlation  function  when  using  edge  content 
quantization  for  TV-to-IR  correlation.  The  most  successful  correla- 
tions were  achieved  with  an  equal  one/zero  reference  image  distribu- 
tion. It  is  recommended  that  the  gradient  threshold  for  the  reference 
image  be  set  at  the  median  gradient  value  of  the  HR-TV  image. 

7.  From  the  set  of  simulations  conducted  for  TV-to-IR  correlations  using 
the  edge  detector  preprocessors  it  is  felt  that  the  3x3  edge 
detection  algorithm  produces  correlation  results  that  are  more 
reliable  than  those  produced  using  the  2x2  edge  detector. 

8.  Contract  personnel  are  aware  of  one  LSI  chip  which  accomplishes  the 
function  of  a Sobel  edge  detector  (3x3  detector).  It  is  recommended 
that  this  chip  or  a similar  chip  be  used  in  a hardware  implementation 
of  an  edge  detection  preprocessing  algorithm. 
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9.  The  simple  edge  detection  algorithms  studied  via  simulation  did  lead 
to  correct  correlation  most  of  the  time.  Other,  more  elaborate, 
edge  detection  systems  would  probably  lead  to  a more  reliable 
correlation  function,  at  the  cost  of  slower  computational  time  and 
increased  hardware  mainly  in  memory.  These  algorithms  were  not 
seriously  considered  in  this  study,  because  of  the  requirement  to 
integrate  the  recommendation  into  an  existing  correlator.  It  is 
recommended  that  additional  study  be  conducted  to  investigate  more 
elaborate  preprocessing  algorithms. 

10.  The  proper  selection  of  gradient  threshold  in  the  edge  detector 
preprocessors  for  both  the  PTS  and  sensor  image  is  critical.  Item  6 
above  outlines  a method  of  threshold  selection  for  the  PTS  image. 
Adjustment  of  the  threshold  for  the  sensor  image  could  be  done  over 
3 or  4 successive  fields  to  select  the  one  which  gives  the  best 
results.  Additional  study  should  be  conducted  for  gradient  thresh- 
old selection  for  the  sensor  image  using  a larger  data  base. 

11.  Proper  PTS  image  resolution  reduction  for  both  TV-to-TV  and  TV-to-IR 
correlations  is  critical.  Incorrect  calculation  of  both  the  horizon- 
tal and  vertical  scale  factors  severly  degrade  the  correlator's 
performance. 

12.  The  dynamic  response  of  the  IR  sensors  limit  the  sampling  rate  used 
for  the  IR  video.  Calculations  made  and  simulations  support  the 
fact  that  an  equivalent  sampling  of  2.5  MHz  is  optimum. 
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